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Claims 

1. An information retrieval technique evaluation method characterized in that it is an 
evaluation method for the retrieval efficiency of an information retrieval technique used on an 
information retrieval system that acquires retrieved data that match data to be retrieved given in 
advance from a database, that calculates the retrieval efficiency of respective retrieval techniques 
from an external evaluation value that indicates whether the data in the database match the data to 
be retrieved determined by at least one or more persons, and a value as to whether or not the data in 
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the database match the data to be retrieved determined by the information retrieval technique, and 
that determines the merits of the retrieval technique, 

the number of external values where it is determined by at least one or more persons that 
the data to be retrieved and any one datum in the database match, and the number of external 
values where it is determined that they do not match are each counted, 

if the judgment as to whether or not they agree with the data to be retrieved that is 
determined by the information retrieval technique for the one datum is the same as the judgments 
on the majority side as to whether or not they agree with the retrieval conditions produced by a 
given external evaluation value, a constant value given in advance is used as the data evaluation 
value, and if the same as the minority side, a value obtained with a formula given in advance from 
the number of external evaluation values on the majority and minority sides is used as a one datum 
evaluation value, 

the aforementioned one datum evaluation value is created for the combination of all the 
given data to be retrieved and the data in the database and the average value of all the one datum 
evaluation values is calculated, 

retrieval efficiency is expressed using one index by using said average value as the 
retrieval efficiency evaluation value for the information retrieval technique, 

and the merits of each information retrieval technique are determined by sequencing the 
evaluation value calculated for at least one or more information retrieval techniques. 

2. The information retrieval technique evaluation method mentioned in Claim 1 
characterized in that when the aforementioned one datum evaluation value is calculated, the actual 
evaluation value is calculated from the number of people on the minority side and the number of 
people on the majority side in accordance with a predetermined formula, 

a predetermined number of people is subtracted from the number of people on the majority 
side, the value subtracted from the majority side is added to the number of people on the minority 
side, and an imaginary evaluation value is calculated in accordance with a predetermined formula, 

the difference between the imaginary evaluation value and the actual evaluation value is 
used as the degree of wobble of the one datum evaluation value and the average value of the degree 
of wobble for the combination of all retrieval conditions and the data in the database is calculated, 

the retrieval efficiency with a confidence interval that increases or decreases the value of 
the extent of wobble is calculated for the aforementioned retrieval efficiency value, 

and the merits of the retrieval efficiency of multiple information retrieval techniques are 
judged by calculating the maximum value and the minimum value for the retrieval efficiency with 
said confidence interval and one retrieval efficiency value for each retrieval technique with a 
formula given in advance from the aforementioned data evaluation values. 
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3. The information retrieval technique evaluation method mentioned in Claim 1 or 2 
characterized in that the corresponding number of people that agrees with the data to be retrieved 
and the corresponding number of people that do not agree are compared by the external evaluation 
value in the calculation of the aforementioned one datum evaluation value, and when they are the 
same number, a predetermined value is used as the one datum evaluation value to calculate the 
retrieval efficiency of each retrieval technique. 

4. An information retrieval technique device characterized in that it is an information 
retrieval technique evaluation device which is a retrieval efficiency evaluation device for an 
information retrieval technique used on an information retrieval system that acquires retrieved 
data that matches data to be retrieved given in advance, that calculates the retrieval efficiency of 
respective retrieval techniques from an external evaluation value that indicates whether the data in 
the database match the data to be retrieved determined by at least one or more persons, and a value 
as to whether or not the data in the database match the data to be retrieved determined by the 
information retrieval technique, and that determines the merits of the retrieval technique, 

and that is furnished with: an input means for inputting the result of the judgment as to 
whether or not something should be retrieved from at least one or more persons, 

a counting means that counts the number of people that think something should be 
retrieved and the number of people that think something should not be retrieved from said means 
for input of human judgment result between individual retrieval results and individual data to be 
retrieved, 

a determination means that compares the number of people that think something should be 
retrieved counted by said counting means and the number of people that think it should not be 
retrieved, and that determines that said data to be retrieved should be retrieved when the number of 
people that think it should be retrieved is at or above a predetermined ratio, 

a judgment means that judges whether the judgment that the information retrieval 
technique should retrieve for the aforementioned retrieval conditions and the data to be retrieved 
agrees or does not agree, 

a constant value generation means that generates a predetermined constant when the result 
of said judgment is that there is agreement, 

a penalty value generation means that generates a penalty value according to a 
predetermined formula when the result of judgment by the aforementioned judgment means is that 
there is not agreement, 

an average value calculation means that finds the average value of the constant value and 
the penalty value obtained for the combination of all given retrieval conditions and data to be 
retrieved as an average value, 
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and a sorting means that places the average values obtained for each retrieval technique in 
a prescribed order. 

5. The information retrieval technique evaluation device mentioned in Claim 4 
characterized in that it is furnished with: a calculation means that determines the magnitude in the 
difference of human judgment according to a predetermined formula after the number of people 
that think that the results of human judgment should be retrieved and the number of people that 
think they should not be retrieved are counted, 

a comparison means that compares the number of people that think something should be 
retrieved and the number of people that think it should not be retrieved in order to provide noise to 
the human judgment, 

an addition and subtraction means that subtracts a predetermined constant value from the 
value of the number of people on the majority side and adds a predetermined constant to the 
number of people on the minority side as a result of comparing the number of people by said 
comparison means, 

a recalculation means that again calculates the magnitude of the difference in judgment 
between humans according to a predetermined formula from the number of people that think it 
should be retrieved that is reset by said addition and subtraction means and the number of people 
that think it should not be retrieved, 

a second average value calculation means that finds the average value of the result of 
calculating the magnitude of the difference between the human judgment produced by the /3 
aforementioned recalculation means and the judgment containing noise for the combination of all 
retrieval conditions and retrieval data, 

a minimum and maximum value calculation means that calculates the minimum value and 
maximum value of the average value with a predetermined formula from the average value 
obtained by the aforementioned average value calculation means and the average value obtained 
by the aforementioned second average value calculation means, 

an order calculation means that calculates the respective order of each retrieval technique 
by arranging the average value, minimum value, and maximum value in order from the smallest, 

and an order determination means that uses the order value of each retrieval technique for 
each of the three values of average value, minimum value and maximum value as one value with a 
predetermined formula. 

6. The information retrieval technique evaluation device mentioned in Claim 4 or 5 
characterized in that it is furnished with: a comparison means that compares the number of people 
that think it should be retrieved and the number of people that think it should not be retrieved 
between one datum to be retrieved and one [more] datum to be retrieved, 



and a constant generation means that generates a predetermined value if the result of 
comparison by said comparison means is the same number. 

Detailed explanation of the invention 
[0001] 

Industrial application field 

This invention relates to an information retrieval technique evaluation method and device 
for same for automatically selecting the optimal information retrieval technique when multiple 
information retrieval techniques are available. In particular, it relates to an evaluation method and 
device for a database retrieval technique where the relationship between retrieval conditions and 
data that should be retrieved changes often according to [the needs of the] user, as represented by a 
current events information database, such as for newspaper articles. 

[0002] 
Prior art 

In the past, in evaluating information retrieval techniques, two indices, called the recall 
ratio represented by Formula (1) below and the relevance factor represented by Formula (2), would 
have been judged in combination. (There are many, for example 
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Formula (1) 
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Key: 1 



Recall ratio 

Data that people think matches retrieved data 

Data judged to match the retrieved data by the retrieval technique 

Data that people think matches the retrieval conditions 
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3 
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Formula (2) 




Key: 1 Relevance factor 

2 Data that people think matches the retrieval conditions 

3 Data judged to match the retrieval conditions by the retrieval technique 

4 Data judged to match the retrieval conditions by the retrieval technique 

[0003] 

Problems to be solved by the invention 

However, with indices produced by the recall ratio and the relevance factor, the numerator 
values are the same and the difference in denominator value only affects the indices. The 
numerator value with the recall ratio is the number of data that should be retrieved based on human 
judgment, and in the case of the relevance factor, it is the number of data that should be retrieved in 
the judgment of the retrieval technique. When the retrieval technique conditions are relaxed to 
increase the recall ratio without changing human judgment, many results will be retrieved, so the 
relevance factor drops. Conversely, when the relevance factor is increased, the retrieval technique 
conditions will become more strict, so there is a tradeoff relationship in recall ratio and relevance 
factor. For this reason, when the merits and disadvantages of various retrieval techniques are 
evaluated, which technique is suitable could not be judged by whether a technique with higher 
relevance factor or a technique with higher recall ratio should be used. 

[0004] 

In particular, the relevance factor uses a "value produced by the retrieval technique" for the 
denominator. So for example, even if the same value of 30% is used, in a case where 9 are correct 
in the judgment of humans, 30 data are obtained as the retrieved result and all 9 that humans think 
should be retrieved are included, and a case where only 10 data are retrieved and 3 correct data are 
included, it is not possible to judge which case represents the better retrieval technique. 

[0005] 

In the recall ratio, too, even with retrieval from the same retrieved data by a person who is 
retrieving, such as for current events information, when the content of the retrieval results they 



want are different, the denominator of the recall factor changes, and the value of the recall factor 
changes, which would be a problem. 

[0006] 

This problem, when the retrieval efficiency evaluation method is represented as in Figure 2, 
is caused by the fact that the data that a person thinks should be retrieved only receives attention 
when it should be retrieved in response to the information retrieval technique, and the fact that the 
human judgment shifts and data that should not be retrieved are not retrieved is not included in the 
evaluation technique. 

[0007] 

The purpose of this invention, in consideration of the aforementioned problems, is to 
provide an information retrieval technique evaluation method and device for same that can 
accurately judge the optimal information retrieval technique. 

[0008] 

Means to solve the problems 

With this invention, with Claim 1 for achieving the aforementioned purpose, an 
information retrieval technique evaluation method is proposed that is an evaluation method for the 
retrieval efficiency of an information retrieval technique used on an information retrieval system 
that acquires retrieved data that match data to be retrieved given in advance from a database, that 
calculates the retrieval efficiency of respective retrieval techniques from an external evaluation 
value that indicates whether the data in the database match the data to be retrieved determined by 
at least one or more persons, and a value as to whether or not the data in the database match the 
data to be retrieved determined by the information retrieval technique, and that determines the 
merits of the retrieval technique, the number of external values where it is determined by at least 
one or more persons that the data to be retrieved and any one datum in the database match, and the 
number of external values where it is determined that they do not match are each counted, if the 
judgment as to whether or not the data to be retrieved that is determined by the information 
retrieval technique for the one datum is matched is the same as the judgments on the majority side 
as to whether or not retrieval conditions produced by a given external evaluation value are matched, 
a constant value given in advance is used as the data evaluation value. If it is the same as the 
minority side, a value obtained with a formula given in advance from the number of external 
evaluation values on the majority and minority sides is used as a one datum evaluation value. The 
aforementioned one datum evaluation value is created for the combination of all the given data to 
be retrieved and the data in the database and the average value of all the one datum evaluation 
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values is calculated. Retrieval efficiency is expressed using one index by using said average value 
as the retrieval efficiency evaluation value for the information retrieval technique, and the merits 
of each information retrieval technique are determined by sequencing the evaluation value 
calculated for at least one or more information retrieval techniques. 

[0009] 

And with Claim 2, in the information retrieval technique evaluation method mentioned in 
Claim 1, an information retrieval technique evaluation method is proposed such that when the 
aforementioned data evaluation value is calculated, the actual evaluation value is calculated from 
the number of people on the minority side and the number of people on the majority side of the 
external evaluation value according to a predetermined formula, a predetermined number of 
people is subtracted from the number of people on the majority side, the value subtracted from the 
majority side is added to the number of people on the minority side, and an imaginary evaluation 
value is calculated according to a predetermined formula. The difference between the imaginary 
evaluation value and the actual evaluation value is used as the extent of wobble of the one datum 
evaluation value. The average value of the extent of wobble for the combination of all the retrieval 
conditions and the data in the database is calculated and retrieval efficiency with a confidence 
interval where the extend of wobble in the value of the aforementioned retrieval efficiency value is 
increased and decreased is calculated. The merits of retrieval efficiency of multiple information 
retrieval techniques are judged by calculating one retrieval efficiency value for each retrieval 
technique with a predetermined formula from the maximum value and minimum value of said 
retrieval efficiency with confidence interval and the aforementioned data evaluation value. 

[0010] 

And with Claim 3, in the information retrieval technique evaluation method mentioned in 
Claim 1 or 2, an information retrieval technique evaluation method is proposed where, in the 
calculation of the aforementioned data evaluation value, the number of corresponding people that 
agree with the retrieval data and the corresponding number of people that do not agree are 
compared by the external evaluation value, and when they are the same, the retrieval efficiency of 
each retrieval technique is calculated with a predetermined value as a one datum evaluation value. 

[0011] 

Also, with Claim 4, an information retrieval technique evaluation device is proposed using 
an information retrieval technique used on an information retrieval system that acquires retrieved 
data that match data to be retrieved given in advance, and is an information retrieval technique 
evaluation device that calculates the retrieval efficiency of respective retrieval techniques from an 
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external evaluation value that indicates whether the data in the database match the data to be 
retrieved determined by at least one or more persons, and a value as to whether or not the data in 
the database match the data to be retrieved determined by the information retrieval technique, and 
that determines the merits of the retrieval technique. There are furnished: an input means for 
inputting the result of the judgment as to whether or not something should be retrieved from at 
least one or more persons, a counting means that counts the number of people that think it should 
be retrieved and the number of people that think it should not be retrieved from said means for 
input of human judgment result between individual retrieval results and individual data to be 
retrieved, a determination means that compares the number of people that think something should 
be retrieved counted by said counting means and the number of people that think it should not be 
retrieved, and that determines that said data to be retrieved should be retrieved when the number of 
people that think it should be retrieved is at or above a predetermined ratio, a judgment means that 
judges whether the judgment that the information retrieval technique should retrieve for the 
aforementioned retrieval conditions and the data to be retrieved agrees or does not agree, a 
constant value generation means that generates a predetermined constant when the result of said 
judgment is that there is agreement, a penalty value generation means that generates a penalty 
value according to a predetermined formula when the result of judgment by the aforementioned 
judgment means is that there is not agreement, an average value calculation means that finds the 
average value of the constant value and the penalty value obtained for the combination of all given 
retrieval conditions and data to be retrieved as an average value, and a sorting means that places 
the average values obtained for each retrieval technique in a prescribed order. 

[0012] 

Also, with Claim 5, an information retrieval technique evaluation device is proposed such 
that, in the information retrieval technique evaluation device mentioned in Claim 4, there are 
furnished: a calculation means that calculates the magnitude in the difference of human judgment 
according to a predetermined formula after the number of people that think that the results of 
human judgment should be retrieved and the number of people that think they should not be 
retrieved are counted, a comparison means that compares the number of people that think it should 
be retrieved and the number of people that think it should not be retrieved in order to provide noise 
to the human judgment, an addition and subtraction means that subtracts a predetermined constant 
value from the value of the number of people on the majority side and adds a predetermined 
constant to the number of people on the minority side as a result of comparing the number of 
people by said comparison means, a recalculation means that again calculates the magnitude of the 
difference in judgment between humans according to a predetermined formula from the number of 
people that think it should be retrieved that is reset by said addition and subtraction means and the 
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number of people that think it should not be retrieved, a second average value calculation means 
that finds the average value of the result of calculating the magnitude of the difference between 
the human judgment produced by the aforementioned recalculation means and the judgment 
containing noise for the combination of all retrieval conditions and retrieval data, a minimum and 
maximum value calculation means that calculates the minimum value and maximum value of the 
average value with a predetermined formula from the average value obtained by the 
aforementioned average value calculation means and the average value obtained by the 
aforementioned second average value calculation means, an order calculation means that 
calculates the respective order of each retrieval technique by arranging the average value, 
minimum value, and maximum value in order from the smallest, and an order determination means 
that uses the order value of each retrieval technique for each of the three values of average value, 
minimum value and maximum value as one value with a predetermined formula. 

[0013] 

In addition, with Claim 6, an information retrieval technique evaluation device is proposed 
where, in the information retrieval technique evaluation device mentioned in Claim 4 or 5, a 
comparison means is furnished that compares the number of people that think it should be 
retrieved and the number of people that think it should not be retrieved, between one datum to be 
retrieved and one [more] datum to be retrieved, as well as a constant generation means that 
generates a predetermined value if the comparison results of said comparison means are the same 
number. 

[0014] 
Operation 

With Claim 1 of this invention, the number of external evaluation values where it is 
determined by at least one or more persons that data to be retrieved and any one datum in a 
database match the data to be retrieved, and the number of external evaluation values where it is 
determined that they do not match are each counted. If the judgment as to whether or not there is 
agreement with the data to be retrieved determined by the information retrieval technique for the 
one datum is the same as the majority side of judgments as to whether or not there is agreement 
with the retrieval conditions based on the given external evaluation values, a constant value given 
in advance is used as the one datum evaluation value. If it is the same as the minority side, a value 
obtained with a formula given in advance from the number of external evaluation values on the 
majority side and the minority side is used as the data evaluation value, and the aforementioned 
one datum evaluation value is created for the combination of all given data to be retrieved and the 
data in the database. In addition, the average value of all the data evaluation values is calculated, 
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and by using this as the retrieval efficiency evaluation value for the information retrieval 
technique, the retrieval efficiency is represented using one index. By sequencing the valuation 
values calculated for at least one or more information retrieval techniques, the merits of each type 
of information retrieval technique are determined. 

[0015] 

Also, with Claim 2, when the aforementioned one datum evaluation value is calculated, an 
actual evaluation value is calculated from the value of the number of people on the minority side 
and the number of people on the majority side of the external evaluation value according to a 
predetermined formula. A predetermined number of people is subtracted from the number of 
people on the majority side, the value subtracted from the majority side is added to the number of 
people on the minority side, and an imaginary evaluation value according to a predetermined 
formulate is calculated. The difference between said imaginary evaluation value and the actual 
evaluation value is used as the extent of wobble of the one datum evaluation value. The average 
value of the extent of wobble for the combination of all retrieval conditions and the data in the 
database is calculated, the retrieval efficiency with a confidence interval where the value of the 
extent of wobble for the aforementioned retrieval efficiency value is further increased or decreased 
is calculated, and one retrieval efficiency value is calculated for each retrieval technique with a 
predetermined formula from the maximum value and minimum value of said retrieval efficiency 
with confidence interval and from the aforementioned one datum evaluation value so that the 
merits of retrieval efficiency of multiple information retrieval techniques are judged. 

[0016] 

Also, with Claim 3, in the calculation of the aforementioned data evaluation value, the 
corresponding number of people that agree with the data to be retrieved and the corresponding 
number of people that do not agree is compared by the external evaluation value. When they are 
the same number, a predetermined value is used as the one datum evaluation value and the 
retrieval efficiency of each retrieval technique is calculated. 

[0017] 

Also, with Claim 4 whether said data to be retrieved should be retrieved or not retrieved is 
determined by the counting means from data for the judgment of at least one person as to whether 
or not something input by an input means should be retrieved. So for judgments as to whether not 
something should be retrieved as judged by a human, the number of people for each is counted, a 
predetermined value forjudging whether or not it should be retrieved and the measured value of 
the ratio of the number of people that think it should or should not be retrieved are compared by a 
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determination means, and it is determined whether said data to be retrieved are data that should be 
retrieved. Next, the judgment result as to whether or not the input information retrieval system 
should be performed and the judgment result as to whether or not something should be retrieved 
determined from human judgment are compared by the judgment means, and it is determined 
whether the judgment result obtained by the retrieval system is in agreement with the judgment 
produced by humans. In addition, when the human judgment and the judgment by the retrieval 
technique agree, a predetermined constant value is given by a constant value generation means. 
When the human and retrieval technique judgments are different, a penalty value is calculated with 
a predetermined formula from the value of the ratio of the various numbers of people for the 
judgments as to whether or not something should be retrieved in the judgment of the humans. For 
example, assuming that the constant value when the judgments agree is 0, the penalty value when 
the do not agree should be a value larger than 0. Because of this, when the judgment by the 
retrieval technique agrees completely with the human judgment as to whether or not something 
should be retrieved, 0 is always given to individual retrieval conditions and to individual data to be 
retrieved. For data when the judgments do not agree, a value larger than 0 is obtained, so by 
calculating the average value of the penalty values with an average value calculation means, the 
evaluation value of retrieval technique will be 0 when the judgments agree perfectly, penalty 
values are used that are larger for retrieval techniques that have more cases where the judgments 
do not agree, and the capability of each retrieval technique is expressed numerically as one 
numerical value. In addition, by sequencing the penalty values calculated, for example, in order of 
the smallest, with a sorting means, the capability of at least one or more retrieval techniques can be 
compared. 

[0018] 

Also, with Claim 5, when a judgment as to whether or not to retrieve is performed using 
individual retrieval conditions and individual data to be retrieved, an actual evaluation value, that 
is, the magnitude of the difference in judgments between people, is calculated with a 
predetermined formula from the ratio of the number of people that think it should be retrieved and 
the number of people that think it should not be retrieved by a calculation means. The number of 
people that think it should be retrieved and the number of people that think it should not is 
compared by a comparison means. Based on said comparison results, a predetermined number of 
people is subtracted from the number of people on the majority side of the number of people who 
think it should be retrieved and the number of people who think it should not be retrieved, and a 
predetermined number of people is added to the minority side by an addition and subtraction 
means. The result is again calculated by a recalculation means in accordance with a predetermined 
formula from the ratio of the changed number of people, an imaginary evaluation value, that is, the 
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magnitude of the difference in human judgments including noise is calculated, and the results of 
judgments by humans that are input and the results with noise included in the judgment are created. 
In addition, the magnitude of the differences of these two judgments is measured with a 
predetermined formula, and when the human judgment is slightly different, how much the penalty 
value will be affected is calculated. The effect of this value on the ratio of the number of people is 
large when the judgment of whether or not something should be retrieved is made by a sufficiently 
large number of people, even when an extremely small number of people move to the majority side 
or the minority side, and the effect of the value on the calculated penalty value is also large. Thus, 
by calculating the magnitude of the difference between the human judgment with noise added and 
the human judgment input, to what extent confidence can be placed in it can be numerically 
expressed. The minimum value and maximum value of the evaluation value are calculated with a 
minimum and maximum value calculation means from the retrieval efficiency evaluation value 
and the value that indicates the degree of confidence. When the degree of confidence is low, the 
retrieval efficiency can be represented by data for a broad range of evaluation values, and when the 
degree of confidence is high, by data for a narrow range of evaluation values. In addition, with an 
order calculation means that sequences the efficiency values, and with an order determination 
means that calculates the order of retrieval efficiency as one value from the order in each value 
with a predetermined formula, between minimum values, maximum values, and evaluation values, 
with an evaluation value with a low degree of confidence, for example, the possibility that the 
maximum value will be very large is high. So within the order of retrieval techniques for each of 
the values of maximum value, minimum value, and evaluation value, it is judged that a retrieval 
technique with a smaller maximum value indicates better retrieval efficiency. The result is that 
even when there is a difference in judgments by people, the merits and disadvantage of one or 
more retrieval techniques can be determined automatically. Because of this, when the results 
produced by an information retrieval technique and human judgment are different, a penalty value 
is calculated from the ratio of people who think something should be retrieved and the ratio of 
people who think it should not be retrieved, but if the value of this penalty is viewed statistically, 
the reliability changes according to the number of people making the judgment. So with the device 
mentioned in aforementioned Claim 4, a judgment as to whether or not something should be 
retrieved must necessarily be made by the same number of people, but with the device mentioned 
in Claim 5 there is no such restriction. 

[0019] 

Also, with Claim 6, when the majority and minority are determined, the number of people 
who think that something should be retrieved and the number of people who think it should not be 
retrieved between one datum to be retrieved and one [more] datum to be retrieved are compared by 
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a comparison means. Based on the results of said comparison, when the number of these people is 
the same, a predetermined constant is generated by a constant generation means, so that an 
evaluation value is calculated even when there is no majority or minority. Because of this, even 
when the number of people judge that it should be retrieved and the number of people that think it 
should not be retrieved are the same, it is possible to judge between data that should be retrieved 
and data that should not be retrieved. 

[0020] 

Application example 

Below, an application example of this invention will be explained with reference to the 
figures. Figure 1 is a block diagram that shows an information retrieval technique evaluation 
device in a first application example of this invention. Said device is constituted from a computer 
that primarily a CPU and from programs. In the figure, (10) is an external evaluation value and 
retrieval technique agreement determination module, (2) is an external evaluation value totaling 
module, (3) is a relational judgment module, (4) is a constant value generation module, (5) is a 
penalty calculation module, (6) is an average penalty calculation module, (7) is an evaluation value 
buffer, and (8) is a sorting module. Note that here, each module is constituted as a processing unit 
of a program (software), but they can also be constituted with hardware. 

[0021] 

External evaluation value and retrieval technique agreement determination module (1) 
compares the judgment as to whether or not something should be retrieved produced by the 
retrieval technique and the judgment given by a human. 

[0022] 

External evaluation value totaling module (2) receives input of the results of human 
judgment made by at least one or more persons and counts the number of people that think it 
should be retrieved and the number of people that think it should not be retrieved. 

[0023] 

Relational judgment module (3) determines whether or not the data should be retrieved 
from the number of people who think it should be retrieved and the number of people who think it 
should not be retrieved. 



[0024] 
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Constant value generation module (4) generates a predetermined example numerical value 
when the judgment produced by the retrieval technique and the human judgment agree as 
determined in external evaluation value and retrieval technique agreement determination 
module (1). 

[0025] 

Penalty calculation module (5) determines the penalty value for the fact that the retrieval 
technique considered a wrong judgment from the ratio of the number of people for whether or not 
something should be retrieved produced by humans when the judgment produced by the retrieval 
technique and the human judgment do not agree. 

[0026] 

Average penalty calculation module (6) finds the average of the penalty values calculated 
for the combination of all the retrieval conditions and the retrieval data for one retrieval technique. 

[0027] 

Evaluation value buffer (7) stores the average values calculated as evaluation values along 
with the retrieval technique. 

[0028] 

Sorting module (8) rearranges the retrieval techniques in evaluation value buffer (7) in 
order of the smaller evaluation values. 

[0029] 

With the application example that is constituted as described above, when the capability of 
a retrieval technique is measured, a table of the information retrieval technique retrieval results 
which is the database retrieval results produced with each type of retrieval technique and an 
external evaluation table that is the result of a judgment as to whether or not something should be 
retrieved made by a human for the same data are externally input in advance. 

[0030] 

Of these input data, the table of information retrieval technique retrieval results, as shown 
in Figure 3, is composed of a technique number for the retrieval technique, an identifier that 
represents the set of each retrieval condition and each datum in the database, and a judgment 
flag that represents whether or not said retrieval technique should retrieve the set represented by 
each identifier. 
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[0031] 

An external evaluation value table, as shown in Figure 4, is composed of an identifier that 
represents the set of each retrieval condition and each datum in the database, and a judgment flag 
that indicates whether or not each person thinks each should be retrieved for the set of retrieval 
conditions and data represented by each identifier. 

[0032] 

For the process in the first application example, as shown in Figure 5, first, as step 11, the 
number of people that think it should be retrieved and the number of people that think it should not 
be retrieved are each counted in external evaluation value totaling module (4) for the data in the 
external evaluation value table that is the result judged by humans. Next, as step 12, it is judged 
whether to be data that should be retrieved or data that should not be retrieved from the ratio of the 
number of people that think it should be retrieved and the number of people that think it should not 
be retrieved in relational judgment module (3). The value of the judgment result is sent to external 
evaluation value and retrieval technique agreement determination module (1) along with the 
number of people that think it should be retrieved and the number of people who think it should not 
be retrieved, where they are stored in an external evaluation value buffer (shown in Figure 6) 
provided for external evaluation value and retrieval technique agreement determination 
module (1). 

[0033] 

A method forjudging whether or not to retrieve at step 12 described above can be realized 
by comparing the number of people that think something should be retrieved and the number of 
people who think it should not be retrieved and letting the majority side [decide]. For example, in 
the case of AAa009 in Figure 4, the number of people who think it should be retrieved is 5, and the 
number of people who think it should not be retrieved is 2, so it is determined to be data that should 
be retrieved. 

[0034] 

As another method for deciding data to be retrieved at step 12, when the judgment from 
humans is split, such as when the number of people that think AAa002, for example, should be 
retrieved is 4, and the number of people who think it should not be retrieved is 3, there is the 
possibility that a large amount of meaning information will be provided. So assuming that such 
data should not be retrieved, and for example, setting the threshold at 70% beforehand, when 
following Formula (3) is satisfied, 
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Formula 2 




" •♦♦(81 



Key: 1 
2 
3 



Number of people who think it should be retrieved 
Number of people who think it should be retrieved 
Number of people who think it should not be retrieved 



[the method] can be realized by determining that it is data that should be retrieved. 



[0035] 



Next, data in an information retrieval technique retrieval result table given for individual 
retrieval conditions and individual data in a database are sent to external evaluation value and 
retrieval technique agreement determination module (1) as step 13 and stored in sequence in an 
evaluation value storage buffer in the module. The data input from the information retrieval 
technique retrieval result table and the data sent from relational judgment module (3) are stored 
together as shown in Figure 6 in the evaluation value storage buffer. When two or more retrieval 
techniques are evaluated, the input data from the information retrieval technique retrieval result 
table are replaced and processed in sequence. Next, as step 14, the results of the retrieval technique 
for each retrieval condition/data identifier and the results from relational judgment module (3) are 
compared in external evaluation value and retrieval technique agreement determination module 
(1). When the two values are equal, a signal is sent to constant value generation module (4) as step 
15, and a value of 0, for example, is generated and sent to average penalty calculation module (6). 
When the results of comparison at step 14 is that the retrieval technique and the value of the 
determination from the relational judgment module are different, the number of people that think it 
should be retrieved and the number of people who think it should not be retrieved in the evaluation 
value storage buffer are sent to penalty calculation module (5). As step 16, the penalty when the 
information retrieval technique and the value of relational judgment module (3) are different is 
calculated by a predetermined formula in penalty calculation module (5). 



One actual method of penalty calculation at step 16 can be realized by generating a 
predetermined constant value. For example, in the case of AAaOOl where the value of the 
information retrieval technique retrieval result and the value from the relational judgment module 
are different, as shown in Figure 6, 1 is generated as the constant value, for example, by step 16 
and this value is sent to average penalty calculation module (6). 



[0036] 
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[0037] 

The average value of the values that indicate the degree of matching between the retrieval 
technique and human judgment obtained by constant value generation module and penalty 
calculation module (5) is calculated as step 17 and an evaluation value with the degree of matching 
of human and retrieval technique for the entire retrieval technique is expressed as a numerical 
value. Next, as step 18, the evaluation value is stored in evaluation value buffer (7) along with a 
number assigned to the retrieval technique. 

[0038] 

The processing between step 13 to step 18 if performed for all given retrieval techniques. 
When it is determined that processing for all given retrieval techniques is completed at step 18-1, 
the retrieval technique number and evaluation value are stored in evaluation value buffer (7), and 
they are rearranged in order of smaller evaluation value in sorting module (8) as step 19. 
Rearrangement at step 19 can easily be realized with existing technology, such as insertion, heap 
sort, quick sort, or the like. By outputting the retrieval technique numbers arranged in order of 
smaller evaluation value as step 20, they can be selected in order from the retrieval techniques 
closer to human judgment. 

[0039] 

As another method for realizing penalty calculation at step 16, when AAalOO and AAbOOl 
in Figure 6 are compared, in contrast to the fact that the result from the retrieval technique gives a 
different judgment, and despite the fact that nearly all the people think that AAalOO should not be 
retrieved, with AAbOOl, the number of people that think it should be retrieved and the number of 
people that think it should not be retrieved are extremely close, so it cannot be said that the 
judgment by the retrieval technique differs from that of the humans whether to retrieve or 
not. So by changing the penalty value according to the number of people, giving a larger penalty 
when there is great variation from the human opinion, and giving a smaller penalty when the 
human judgment is ambiguous, classification can be accomplished more precisely in order starting 
with the retrieval techniques closer to human judgment. To realize this, for example, penalty 
calculation module (5) would be configured as shown in Figure 7. 

[0040] 

Penalty calculation module (5) shown in Figure 7 is composed of working buffer (21) that 
temporarily stores the number of people that think something should be retrieved and the number 
of people that think it should not be retrieved that are input, comparison module (22) that compares 
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the two data in working buffer (21), division module (23) that divides the compared result using 
the value of the smaller as the numerator and the value of the larger as the denominator, and 
penalty value generation module (24) that uses the value of the divided value subtracted from 1 as 
the penalty value. 

[0041] 

The processing in penalty calculation module (5) is accomplished as follows. As shown in 
Figure 8, the number of people that think it should be retrieved and the number of people that think 
it should not be retrieved that are input are stored in working buffer (21) as step 25. Next, as step 
28, the values of the two data in working buffer (21) are compared in comparison module (22) and 
the compared result is stored with the smaller value in numerator buffer (23b) of division module 
(23) and the larger value in denominator buffer (23a). As step 27, the values in numerator buffer 
(23b) and denominator buffer (23a) are divided. Finally as step 28, the value divided in penalty 
value generation module (24) subtracted from 1 is used as the penalty value. 

[0042] 

With this method of realization, when the judgment by the retrieval technique is different, 
despite the fact that the human judgments nearly all agree, as for AAalOO in Figure 6, 1 - 1 6 = 
0.83 is given. When the human opinions are split as for AAbOOl, 1 - 3 + 4 = 0.25 is given, which is 
a smaller penalty value. The result will be that when the human judgment is split, the penalty value 
for techniques that will give many of the human judgments will be small, and the average penalty 
value for techniques that show different judgments despite the fact that the human judgments agree 
will be larger. So both can be clearly separated. 

[0043] 

The idea of the penalty in this invention is based on the fact that if the result produced by a 
retrieval technique is to give a different judgment, despite the fact that nearly all the humans think 
that it should not be retrieved, as for AAalOO in Figure 6, the degree of importance of the 
information that the retrieval technique does not agree with human judgment is high. If the number 
of people that think it should be retrieved and the number of people that think it should not be 
retrieved is very competitive, as for AabOOl, it cannot be determined unconditionally whether or 
not it should be retrieved, so the importance of the information that there is not agreement with 
human judgment can be considered small. 



[0044] 
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This idea can also be realized with a technique where processing at step 16 applies an 
entropy formula, since there is a portion similar to the concept of information entropy. One 
application example where an entropy formula is applied can be realized by the configuration as 
shown in Figure 9. One example of penalty calculation module (5) shown in Figure 9 is composed 
of working buffer (31) that temporarily stores the number of people that think it should be 
retrieved and the number of people that think it should not be retrieved that are input, comparison 
module (22) that compares the two data in working buffer (31), calculation module (32) that 
calculates the total number of people that made a judgment from the values of the two data in 
working buffer (31), entropy calculation module (33) that calculates the entropy value from the 
number of people that think it should be retrieved, the number of people that think it should not be 
retrieved, and the number of people responding from obtained from number of people responding 
calculation module (32), and penalty value generation module (34) that uses the entropy value 
subtracted from 1 as the penalty value. 



[0045] 

As processing in the penalty calculation module (5) that is configured as described above, 
as shown in Figure 10, as step 35, the number of people that think something should be retrieved 
and the number of people that think it should not be retrieved that are input are stored in working 
buffer (3 1). Next, as step 36, the values for the two data in working buffer (31) are added together 
to calculate the number of people in number of people responding calculation module (32). Next as 
step 37, entropy calculation is performed with Formula (4) shown below in entropy calculation 
module (33). 



[0046] 



Formula 3 




Key: 1 Number of people that think something should be retrieved 
2 Solved number of people 



(1-E) is calculated for entropy (E) found here in penalty value generation module (34) as step 38 to 
accomplish penalty calculation. With this method of realization, when the retrieval technique 
judgment differs, despite the fact that the judgments of the humans nearly all agree as for AAalOO 
in Figure 6, a small penalty value of 1 - 0.99 = 0.01 will be given. The result is that the penalty 
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value for a technique that differs from many human judgments when the human judgments are 
split will be smaller. With a technique that shows a different judgment, despite the fact that the 
human judgments nearly all agree, the average penalty value will be larger, so both can be clearly 
separated. 

[0047] 

Next, a second application example of this invention will be explained. Figure 1 1 is a block 
diagram of an information retrieval technique evaluation device in a second application example. 
Said device is composed of a computer, which is primarily a CPU, and programs. 

[0048] 

With this second application example, as shown in Figure 11, nearly the same 
configuration as the first application example shown in Figure 1 is used. In this application 
example, in the information retrieval technique retrieval result table, which is the retrieval result 
for a database produced by each type of retrieval technique, and an external evaluation value table, 
which is the result of judgment by humans as to whether or not the same data should be retrieved, 
are externally input. 

[0049] 

To measure the capability of a retrieval technique from the input data, the information 
retrieval technique evaluation device in the second application example is composed of 10 
modules: external evaluation value and retrieval technique agreement determination module (1) 
that compares the judgment as to whether or not something should be retrieved produced by the 
retrieval technique and the judgments given by humans, external evaluation value totaling module 
(2) into which the result of human judgment made by at least one or more persons is input and that 
totals the number of people that think something should be retrieved and the number of people that 
think it should not be retrieved, (3) relational judgment module (3) that determines whether or not 
something should be retrieved from the ratio of the number of people that think it should be 
retrieved and the number of people that think it should not be retrieved, constant value generation 
module (4) that generates a predetermined constant value when the judgment produced by the 
retrieval technique and the human judgment agree as determined by external evaluation value and 
retrieval technique agreement determination module (1), penalty calculation module (5) that 
determines a penalty value for the fact that the retrieval technique has made an incorrect judgment 
from the ratio of people that think it should or should not be retrieved when the judgment produced 
by the retrieval technique and the human judgment do not agree, average penalty calculation 
module (6) finds the average of the penalty values calculated for combination of all the retrieval 
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conditions and data to be retrieved for one retrieval technique, actual evaluation value calculation 
module (41) that calculates the amount of information for human judgments from the ratio of 
people that think something should or should not be retrieved, imaginary evaluation value 
calculation module (42) that calculates the amount of information for human judgments for when 
human judgment changes slightly, intermediate buffer (43) that stores the calculated average 
penalty value and the actual evaluation value and imaginary evaluation value as a set with the 
retrieval technique, and technique order calculation module (44) that sequences the retrieval 
technique in intermediate buffer (43) from the evaluation value, actual evaluation value, and 
imaginary evaluation value. Here too, the same as in the first application example, each module is 
also constituted as a program (software) processing unit, but they can also be constituted with 
hardware. 

[0050] 

The information retrieval technique retrieval result table and the external evaluation value 
table, which are input data, are each the same as in the first application example described above, 
so they will be omitted. However, in the first application example, there was only one external 
evaluation value table for all the retrieval techniques being evaluated, but in this second 
application example, an external evaluation value table with a different number of people 
performing evaluation can also be used for each evaluation technique. 

[0051] 

As the processing in the second application example, as shown in Figure 2, first, as step 51, 
for the data in the external evaluation value table, which is the result of judgment by humans, the 
number of people that think it should be retrieved and the number of people that think it should not 
be retrieved are each counted in eternal evaluation value totaling module, and [this] is sent to 
relational judgment module (3) and to actual evaluation value calculation module (41) and 
imaginary evaluation value calculation module (42). 

[0052] 

Next as step 52, it is determined in relational judgment module (3) whether it is data that 
should be retrieved or data that should not be retrieved, from the ratio of the number of people that 
think it should be retrieved and the number of people that it should not be retrieved. The value for 
the result of the judgment is sent to external evaluation value and retrieval technique agreement 
determination module (1) along with the total of people that think it should be retrieved and the 
people that think it should not be retrieved, where it is stored in the external evaluation value buffer 
shown in Figure 6. 
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[0053] 

The method forjudging whether or not something should be retrieved at step 52 is the same 
method as used in the first application example, so an explanation will be omitted. 

[0054] 

Next, data in the conditional retrieval technique retrieval result table given for individual 
retrieval condition and individual data in the database are sent to external evaluation value and 
retrieval technique agreement determination module (1) as step 53, where they are stored in order 
in an evaluation value storage buffer in the module. The evaluation value storage buffer stores data 
input from the information retrieval technique retrieval result table and data sent from relational 
judgment module (3) in pairs as shown in Figure 4. When two or more retrieval techniques are 
evaluated, data input from the retrieval technique retrieval result and the external evaluation value 
table are replaced and processed in sequence. Next, as step 54, the results of the retrieval technique 
for each retrieval condition and data identification and the results from relational module (3) are 
compared in the external evaluation value and retrieval technique determination module. When the 
two values are equal, as step 55, a signal is sent to constant value generation module (4), a value of 
0, for example, is generated, and it is sent to average penalty calculation module (6). 

[0055] 

When the result of comparison at step 54 is that the value determined by the retrieval 
technique and by relational judgment module (3) are different, the value for the number of people 
that think it should be retrieved and the value for the number of people that think it should not be 
retrieved in the evaluation value storage buffer are sent to penalty calculation module (5). As step 
56, a penalty for when the information value from the retrieval technique and from relational 
judgment module (3) are different is calculated with a predetermined formula in penalty 
calculation module (5). 

[0056] 

One method of realizing the penalty calculation at step 56 is to use the technique using 
entropy in the first application example, for example, so an explanation will be omitted here. 

[0057] 

For a value that represents the degree of agreement between retrieval technique and human 
judgments obtained by constant value generation module (4) and penalty calculation module (5), 
the average value is calculated as step 57, and an evaluation value is generated to express the 
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degree of agreement between human and retrieval technique judgments for the entire retrieval 
technique. 

[0058] 

In actual evaluation value calculation buffer (41), as step 58, the amount of information is 
calculated from the number of humans that think it should be retrieved and the number of humans 
that think it should not be retrieved. Calculation of the amount of information can be accomplished 
with the same technique as calculation of entropy in the first application example, 
for example, so an explanation will be omitted, since it can easily be analogized. 

[0059] 

In imaginary evaluation value calculation buffer (42), as step 59, the majority side of the 
number of people that think something should be retrieved and the number of people that think it 
should not be retrieved is reduced by a predetermined number of people, for example, 1 person, 
and the number of people reduced is added to the minority side. So noise can be added to the result 
of human judgment. 

[0060] 

Next, as step 60, the amount of information for a case where noise is provided is calculated 
to give an imaginary evaluation value. Calculation of the imaginary evaluation value is the same as 
the calculation of entropy in the first application example, so an explanation will be omitted here. 
Processing in actual evaluation value calculation module (41) and imaginary evaluation value 
calculation module (42) is performed for all cases as is apparent from Figure 11. 

[0061] 

Next, as step 61, the evaluation value, actual evaluation value, and imaginary evaluation 
value are stored in intermediate buffer (43) along with a number assigned to the retrieval 
technique. 

[0062] 

The processing between step 51 to step 61 is performed for all the given retrieval 
techniques. When it is determined at step 61-1 that processing for all given retrieval techniques is 
completed, the retrieval technique number and evaluation value are stored in intermediate buffer 
(43). As step 62, the order of the retrieval techniques in the order of similarity to human judgment 
is determined in technique order calculation module (44) from the evaluation value and the actual 
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evaluation value and imaginary evaluation value. In addition, as step 63, the technique retrieval 
numbers are output in order of the smaller evaluation value. 



Technique order evaluation module (44) is realized with a configuration as shown in 
Figure 13, for example. Technique order calculation module (44) is constituted from 4 modules: 
confidence interval calculation module (71) that calculates a confidence interval from the three 
values of evaluation value, actual evaluation value and imaginary evaluation value sent from 
intermediate buffer (43), sequencing module (72) that sequences the retrieval techniques relative 
to the evaluation value and calculated lower and upper limits of each, synthesizing uniform 
calculation module (73) that generates one order value according to a predetermined formula from 
the respective order of the lower limit, upper limit and evaluation value for individual retrieval 
techniques assigned by sequencing module (72), and sorting module (74) that arranges and outputs 
the order value calculated for each retrieval technique in order from the smallest. 



As processing in technique order calculation module (44), as shown in Figure 14, as step 
81, for example, the lower limit value and upper limit value are calculated using Formulas (5) and 
(6) below in confidence interval calculation module (71) into which are input the evaluation value 
and the actual evaluation value and imaginary evaluation value from intermediate buffer (43). 



[0063] 



[0064] 



[0065] 
Formulas 4 




Key: 1 



Lower limit value 
Evaluation value 



2 
3 
4 
5 



Actual evaluation value 
Imaginary evaluation value 
Upper limit value 



The calculated upper limit and lower limit value and the evaluation value are stored in a buffer in 
sequencing module (72) as step 82. Steps 81 and 82 are repeated until calculation of the upper limit 
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value and lower limit value for all the retrieval techniques is completed. After this, it is determined 
whether calculation of the upper limit value and lower limit value has been completed for all the 
retrieval techniques at step 80. After calculation is completed, as step 83, a number is assigned to 
each technique in order from the smallest for each of the lower limit value, upper limit value, and 
evaluation value in alignment and order determination part (72a) in sequencing module (72). This 
processing can easily be realized with a general sorting algorithm, so a detailed explanation will be 
omitted. 

[0066] 

Next, as step 84, a synthesized order is calculated from the lower limit order, evaluation 
order, and upper limit order in synthesized order value calculation module (73). The calculation 
method for synthesized order can be found by totaling each order as shown in the figure of the 
buffer in synthesized order calculation module (73) in Figure 13, for example. 

[0067] 

For the value for each of the lower limit order and upper limit order, the lower limit order 
represents retrieval efficiency when viewed optimistically, and the upper limit value represents 
retrieval efficiency when viewed pessimistically. Here, as another calculation method for the 
synthesized order value, if a pessimistic viewpoint is taken for the retrieval technique so that 
techniques with a larger upper limit value are discarded, when the total of the lower limit order, the 
upper limit order, and the evaluation order is found, for example, the value of the upper limit order 
is added two more times. Then, in the case of Figure 13, technique A will be 13, technique B 8, 
technique C 9, and technique D 20, and the order value of technique B will be the smallest. 

[0068] 

Next, as step 85, technique numbers are output in order of the smaller synthesized order 
values in sorting module (74). Alignment processing of the synthesized order values in sorting 
module (74) can easily be accomplished with a general sorting algorithm, so a detailed explanation 
will be omitted. 

[0069] 

Next, a third application example of this invention will be explained. With the third 
application example, the same configuration can be added to both the first and the second 
application examples described above, so here it will be explained using the first application 
example. 



27 



[0070] 

A block diagram in the third application example can be represented as shown in Figure 15. 
In the configuration shown in Figure 15, same number check module (9) and same number 
occurrence constant generation module (10) are added to the configuration of the first application 
example. 

[0071] 

For processing in the third application example, as shown in Figure 16, the number of 
people that think it should be retrieved and the number of people that think it should not be 
retrieved are compared at same number check module (9) to determine whether or not they are the 
same number as step 91 between step 1 1 and step 21 in Figure 5. Next when the number of people 
is the same number, as step 92, -1, for example, is sent to external evaluation value and 
retrieval technique agreement determination module (1), along with the number of people that 
think it should be retrieved and the number of people that think it should not be retrieved, in place 
of the results from relational judgment module (9) as impossible to determine. 

[0072] 

When the number of people in the results for determining whether or not something should 
be retrieved is not the same, step 12 is executed. Processing in steps 12 and 13 are the same as the 
first application example, so an explanation will be omitted. 

[0073] 

Next, in place of step 14 shown in Figure 5, as step 93, the results of the retrieval technique 
for each retrieval condition and data identifier and the results of relational judgment module (3) are 
compared in external evaluation value and retrieval technique agreement determination module 
(1). When the two values are equal, or determination is impossible, for example, when -1 is given 
to the determination flag produced by relational judgment module (3) in the buffer, as step 15, a 
signal is sent to same number occurrence constant generation module (10), a value of 0, for 
example, is generated, and it is sent to average penalty calculation module (6). Processing after 
step 15 is the same as the first application example, so an explanation will be omitted. 

[0074] 

When the same function is added to the technique in the second application example, it can 
be achieved by adding same number check module (9) between external evaluation value totaling 
module (2) and relational judgment module (3) in Figure 1 1 in the same way. Processing when it is 
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added is the same as when same number check module (9) is added to the configuration of the first 
application example, and an explanation will be omitted, since it can easily be analogized. 

[0075] 

An example of data in the buffer in external evaluation value and retrieval technique 
agreement determination module (1) in the third application example is as shown in Figure 17. 

[0076] 

Next, the results obtained when the capabilities of this invention were tested are shown 

below. 

Experiment conditions: 

Number of data to be retrieved used for the experiment: 12 (retrieval objects are composed 
of a set of keywords accompanying newspaper articles selected at random, and the nouns, proper 
nouns, and verbal nouns comprising the newspaper article text) 

Number of samples of retrieved data actually used: 50 for each condition (randomly 
extracted) 

Retrieval techniques compared: 4 

1. AND retrieval: retrieved data acquired with all keywords in the retrieval object 

[0077] 

2. OR retrieval: retrieved data acquired with any one or more of the keywords in the 
retrieval object. 

[0078] 

3. Best retrieval: Retrieval performed artificially so that both the relevance factor and recall 
ratio will be high for evaluation by combining any one or more keywords in the retrieval object. 

4. Single word retrieval: data with a high degree of matching to single words in the set of 
nouns, proper nouns and verbal nouns comprising the text of the retrieval object is acquired. 

[0079] 

Number of test subjects: 9 

Number of extended one evaluation values: 12 x 50 = 600. 

Calculation methods in the evaluation: The technique where the concept of entropy was applied in 
the first through third application examples was used. With this experimental example, the number 
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of test subjects is an odd number, and the judgments made by human are the same, so the second 
and third application examples will be the same as the first application example. This is because 
the width between the upper limit value and the lower limit value in the second application 
example is the same for any technique, and the human judgments as to whether or not something 
should be retrieved in the third application example will not be the same number. 



[0080] 

Results: the results when the experiment was performed are shown below. The relationship 
between recall ratio and relevance factor used in the past and the evaluation value used in this 
invention is reflected in the following table. 



[0081] 
Table 1 
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55.3% 
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0. 075 



Key: 1 Retrieval technique 

2 Recall ratio 

3 Relevance factor 

4 This invented technique 

5 AND retrieval 

6 OR retrieval 

7 Best retrieval 

8 Single word retrieval 



As is clear from the table, using just the recall ratio and relevance factor, it cannot be 
determined whether OR retrieval is better, or best retrieval is better, or single word retrieval is 
better. However, in the case of OR retrieval, data that should not be retrieved in the judgment of 
the most people was included. With this invented technique, there is clearly a difference between 
OR retrieval, best retrieval, and single word retrieval. In addition, it also shows that the best 
retrieval that performed a retrieval so that an artificially high retrieval efficiency would occur is the 
optimum method. The result is that, with this invented technique, it was possible to determine the 
merits and disadvantages of the retrieval techniques to be, in order starting with the best retrieval 
technique, best retrieval, single word retrieval, AND retrieval, and OR retrieval. 
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[0082] 

Effect of the invention 

As explained above, with the information retrieval technique evaluation method mentioned 
in Claim 1 of this invention, in a database retrieval technique for acquiring data that matches the 
retrieval conditions from a database by inputting the retrieval conditions, by inputting judgment 
data for whether or not retrieval conditions are matched and data for the retrieval results obtained 
by each retrieval technique for retrieval conditions provided at least one or more humans and for 
each datum in the database or sample data extracted from the database, the merits and 
disadvantages of the retrieval efficiency of at least one or more retrieval techniques can be 
determined automatically. 

[0083] 

Also, with Claim 2, in addition to the effect described above, a difference occurs for the 
result of human judgment in the reliability of values viewed statistically when judgments are made 
by a large number of people and when judgments are made by a small number of people, e.g., 1 or 
2. The merits of a retrieval technique can be determined even though the people who are the test 
subjects in the human based evaluation differ regarding the retrieval technique. 

[0084] 

Also, with Claim 3, in addition to the effects described above, even if the number of people 
that make a judgment is an even number and the number of people that think it should be retrieved 
and the number or people that think it should not be retrieved is the same, by comparing the 
number of people that think it should be retrieved and the number of people that think it should not 
be retrieved, between one datum to be retrieved and one [other] datum to be retrieved, and 
generating a predetermined value when they are the same number, the merits of the retrieval 
system can be determined when an evaluation is performed with any number of people. 

[0085] 

Also, with the information retrieval technique evaluation device mentioned in Claim 4, in a 
database retrieval technique for acquiring data that matches the retrieval conditions from a 
database by inputting the retrieval conditions, by inputting judgment data for whether or not 
retrieval conditions are matched and data for the retrieval results obtained by each retrieval 
technique for retrieval conditions provided by at least one or more humans and for each datum in 
the database or sample data extracted from the database, the merits and disadvantages of the 
retrieval efficiency of at least one or more retrieval techniques can be determined automatically. 



31 



[0086] 

Also, with Claim 5, in addition to the effect described above, a difference occurs for the 
result of human judgment in the reliability of values viewed statistically when judgments are made 
by a large number of people and when judgments are made by a small number of people, e.g., 1 or 
2. The merits of a retrieval technique can be determined even though the people who are the test 
subjects in the human based evaluation differ regarding to the retrieval technique. 

[0087] 

Also, with Claim 6, in addition to the effects described above, even if the number of people 
that make a judgment is an even number and the number of people that think it should be retrieved 
and the number or people that think it should not be retrieved is the same, by comparing the 
number of people that think it should be retrieved and the number of people that think it should not 
be retrieved, between one datum to be retrieved and one [other] datum to be retrieved, and 
generating a predetermined value with a constant generation means when they are the same 
number, the merits of the retrieval system can be determined when an evaluation is performed with 
any number of people. 

Brief description of the figures 

Figure 1 is a system block diagram of an information retrieval technique evaluation device 
in a first application example of this invention. 

Figure 2 is a conceptual diagram that shows an evaluation method for retrieval efficiency. 

Figure 3 is a figure that shows the data structure and data examples in the information 
retrieval technique retrieval result table in the first application example. 

Figure 4 is a figure that shows the data structure and data examples in the external value 
evaluation table in the first application example. 

Figure 5 is a flowchart that shows processing in the information retrieval technique 
evaluation method in the first application example. 

Figure 6 is a figure that shows the data structure in the external evaluation value buffer in 
the first application example. 

Figure 7 is a system block diagram of the penalty calculation module in the first application 
example. 

Figure 8 is a flowchart that shows the flow of processing by the penalty calculation module 
in the first application example. 

Figure 9 is a system block diagram that shows another example of the penalty calculation 
module. 
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Figure 10 is a flowchart that shows the processing flow in the other example of the penalty 
calculation module. 

Figure 1 1 is a system block diagram of an information retrieval technique evaluation 
device in a second application example. 

Figure 12 is a flowchart that shows the processing flow by the information retrieval 
technique evaluation method in the second application example. 

Figure 13 is a system block diagram of the technique order calculation module in the 
second application example. 

Figure 14 is a flowchart that shows the processing in the technique order calculation 
module in the second application example. 

Figure 15 is a system block diagram of an information retrieval technique evaluation 
device in the third application example. 

Figure 16 is a flowchart that shows the processing in the information retrieval technique 
evaluation method in the third application example. 

Figure 17 is a figure that shows the data structure and data examples in the external 
evaluation value buffer in the third application example. 

Explanation of reference symbols 

(1) ... external evaluation value and retrieval technique agreement determination module, (2) ... 
external evaluation value totaling module, (3) . . . relational judgment module, (4) . . . constant 
value generation module, (5) ... penalty calculation module, (6) ... average penalty calculation 
module, (7) . . . evaluation value buffer, (8) . . . sorting module, (9) . . . same number check module, 
(21) ... working buffer, (22) ... comparison module, (23) ... division module, (24) ... penalty 
value generation module, (31) . . . working buffer, (32) . . . number of people responding calculation 
module, (33) ... entropy calculation module, (34) ... penalty value generation module, (41) ... 
actual evaluation value calculation module, (42) imaginary evaluation value calculation module, 
(43) ... intermediate buffer, (44) ... technique order calculation module, (71) ... confidence in- 
terval calculation module, (72) . . . sequencing module, (73) . . . synthesized order value calculation 
module, (74) ... sorting module. 
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Figure 1 
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External evaluation value and retrieval technique agreement determination module 
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External evaluation value totaling module 
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Relational judgment module 
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Constant value generation module 
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Penalty calculation module 
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Average penalty calculation module 
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Sorting module 
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Key: 1 Should be retrieved 

2 Recall ratio 

3 Judgment by human 

4 Wobble in human judgment 

5 Relevance factor 

6 Judgment by information retrieval technique 

7 Should not be retrieved 

8 Rate at which data that should not be retrieved are not retrieved 
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Retrieval technique name 

Retrieval technique number 

Retrieval conditions — data identifier 

Judgment flag 

(0: should not be retrieved) 

1: should be retrieved) 

AND retrieval 

Information retrieval technique retrieval result table 
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Response of test subject 1 
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Response of test subject 2 
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Response of test subject 3 
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Figure 5 



Key: 1 Evaluation start 
2 End 

1 1 Count the number of people that think it should be retrieved and the number of 
people that think it should not be retrieved, for the data for each identifier in ex- 
ternal evaluation value table 
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12 Determine whether it should be retrieved from the number of people that think it 
should be retrieved and the number of people that think it should not be retrieved 

13 Data in information retrieval technique retrieval result table stored in evaluation 
value storage buffer 

14 Retrieval technique result and relational judgment module result the same? 

15 Constant value is generated 

16 One relational penalty is calculated from the number of people that think it should 
be retrieved and the number of people that think it should not be retrieved 

17 Average value (evaluation value) of penalties calculated 

18 Evaluation value stored in evaluation value buffer 
18-1 All techniques input? 

19 Evaluation values rearranged in order of smaller 

20 Retrieval technique number is output. 
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Figure 6 

Key: 1 Retrieval conditions - identifier 

2 Judgment flag produced by information retrieval technique retrieval result table 

3 Judgment flag produced by relational judgment module 

4 Number of people that think it should be retrieved 

5 Number of people that think it should not be retrieved 

6 External evaluation value buffer 
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Penalty calculation module 

Working buffer 

Comparison module 

Division module 

Numerator buffer 

Denominator buffer 

Penalty value generation module 
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Figure 8 

Penalty calculation module 
Module end 

Number of people that think it should be retrieved and number of people that think 
it should not be retrieved stored in working buffer 

Two values compared, smaller value stored in numerator buffer and larger value 
stored in denominator buffer 

Division performed using values in numerator buffer and denominator buffer. 
Result of division subtracted from 1. 
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Key: 1 Penalty calculation module 

31 Working buffer 

32 Number of people responding calculation module 

33 Entropy calculation module 

34 Penalty value generation module 

(T) ( ^^4^m^^-^ ) 
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Key: 1 Penalty value calculation module 

2 Module end 

35 Number of people that think it should be retrieved and number of people that think 
it should not be retrieved are stored in working buffer 

36 Two values are added to calculate number of people responding. 

37 Entropy calculated 

38 Result of entropy subtracted from 1 
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Figure 11 

Key: 1 External evaluation value and retrieval technique agreement determination module 

2 External evaluation value totaling module 

3 Relational judgment module 

4 Constant value generation module 

5 Penalty calculation module 

6 Average penalty calculation module 

10 Information retrieval technique retrieval result table 

1 1 External evaluation value table 

12 Technique number 

13 Evaluation value 

14 Actual evaluation value 

15 Imaginary evaluation value 

41 Actual evaluation value calculation module 

42 Imaginary evaluation value calculation module 

43 Intermediate buffer 

44 Technique order calculation module 
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Key: 1 
2 



Evaluation start 
End 
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51 Number of people that think it should be retrieved and number of people that think 
it should not be retrieved counted for data for each identifier in external evaluation 
value table 

52 Determine whether it should be retrieved from the number of people that think it 
should be retrieved and number of people that think it should not be retrieved 

53 Data in information retrieval technique retrieval result table stored in evaluation 
value storage buffer. 

54 Result of retrieval technique and result of relational judgment module the same? 

55 Constant value generated 

56 One relational penalty calculated from the number of people that think it should be 
retrieved and number of people that think it should not be retrieved 

57 Average value (evaluation value) of penalties calculated 

58 Amount of information for one relation calculated from the number of people that 
think it should be retrieved and number of people that think it should not be re- 
trieved, and the average value for all relations found 

59 Value on the majority side is decreased and added to the minority side to generate 
noise. 

60 Amount of information for one relation is calculated using the number of people 
with noise, and the average value for all relations is found. 

61 Evaluation value stored in evaluation value buffer 
61-1 All techniques input? 

62 Rearranged in order of smaller evaluation value 
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Figure 13 

Key: 1 Lower limit 

2 Evaluation value 

3 Upper limit 

4 Technique number 

5 Lower limit value 

6 Lower limit order 

7 Evaluation value 

8 Evaluation value order 

9 Upper limit value 

1 0 Upper limit order 

11 Technique [A, B, etc.] 

12 Synthesized order value 

13 Technique order calculation module 

71 Confidence interval calculation module 

72a Arrangement and order determination part 

73 Synthesized order value calculation module 

74 Sorting module 
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Key: 1 Technique order calculation module 
2 End 

80 Processing done for all data evaluation values? 

81 Lower limit value and upper limit value calculated 

82 Stored in buffer in sequencing module 

83 Sequencing performed for lower limit value, upper limit value, and evaluation 
value 

84 Synthesized order calculated 

85 Output in order of smaller synthesized order value. 



12 ] *feff««3C 1 



IvT ft! 



, j 1 A« » 

1 I 





**** 























■^7 



Figure 15 



45 



Key: 1 External evaluation value and retrieval technique agreement determination module 

2 External evaluation value totaling module 

3 Relational judgment module 

4 Constant value generation module 

5 Penalty calculation module 

6 Average penalty calculation module 

7 Evaluation value buffer 

8 Sorting module 

9 Same number check module 

10 Same number occurrence constant generation module 

1 1 Information retrieval technique retrieval result table 

12 External evaluation value table 

13 Technique number 

14 Evaluation value 
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Figure 16 



Key: 1 
2 
11 



12 
13 



Evaluation start 
End 

Number of people that think it should be retrieved and number of people that think 
it should not be retrieved counted for data for each identifier in external evaluation 
value table 

Determine whether it should be retrieved from the number of people that think it 
should be retrieved and number of people that think it should not be retrieved 
Data in information retrieval technique retrieval result table stored in evaluation 
value storage buffer 
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15 Constant value is generated 

16 One relational penalty is calculated from the number of people that think it should 
be retrieved and the number of people that think it should not be retrieved 

17 Average value (evaluation value) of penalties calculated 

18 Evaluation value stored in evaluation value buffer 
18-1 All techniques input? 

19 Rearranged in order of smaller evaluation value 

20 Retrieval technique number is output 

91 Number of people that think it should be retrieved and number of people that think 
it should not be retrieved the same number? 

92 Decision that it is not possible to determine 

93 Result of retrieval technique and result of relational judgment module the same, or 
determination impossible? 
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Figure 17 



Key: 1 Retrieval conditions - identifier 

2 Judgment flag produced by information retrieval technique retrieval result table 

3 Judgment flag produced by relational judgment module 

4 Number of people that think it should be retrieved 

5 Number of people that think it should not be retrieved 

6 External evaluation value buffer 
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[0017] ifc, M#B4*cJ:;mr. H-$l#gfc:J: 

^b^-mknmwmnT-ffrt*. m&m 

bmtf^$i^ijwAmL<mjmb&imztL. m 
mmm.T-?#w%tt^ ■T-fx-b&nfrgfrff 

StegSixS. «Br#at^J:-?-C, A;fr3*i*:tiif8 
S4Wfc-gcLT^S*>#8tS£S*T.S. AOtf> 

Ambm^^mmmupm^x^tm^izit. **** 
^ 4 mmi^mz * 0 j Amcrwmiz&if&tmt^ 

If. Wi^-itL^*^O^fi^0 fc 3*u -Set* 
*^5t*^^^f- 4 Offl* c O J: 0 ^fflt Sit 
& „ iixfcrj: -&-5*^Afl3^Bf^^# 

m0mtf%£iz--$ctzi§ f £tz^ -^—xvmgk 

ftb-r>--^<7)mmML7 : -?tzftLXli!l l z 0 h 
*U WKaMSWtvvf-^fcWLTIi. OiOtA^ 

4 ffi<W«3ffi£ WW S i b J: 0 . 
Zb&X'Zi. 

[0018] n$3H5(=J:nif . 



(6 

9 

% t itz/Mtmmt^x^t uzA^^mcr>m 

&bt>tvrzAm i mt>tii>. Z<m&$3zZtitzA®. 
Aiao«rc^^£§#tt«3ivC. AXZtitzA 

isti &$mcr>t!i3ktvm£sj xam^izt&%ti : &sR 
0msi<m^<7>A-zzmfezti. Afstvmff&Lm 10 

m>m<?>XW?%$lcDMfr'bf!lcr>®*3$ -> tz blXi>. 

A$wm£izi§-z_&&>gi£k$ < . sm.znz't.-r)^ 

4^<7)&*gcr)mi>AZ\,\ i.r>XJ4Xm-th1xtzA 
-ThZbX\ b'nW&{m : ti3V?>t>*iJ t %MttX'Z&. 

mmmomsmb. ms^msEt^rm^^ &>m 
Awx-^mzx *)mmm<rm/m. &xmmmz 20 

t*m\ ^%&\z\$m mmm<r>u<r>mMT- 9 t Lxmn 
wmzm-z b & . s &>i>.m±&vmA 

frh-^<m.b Lxmmmnimm&tmmfe 
^mziL*). miKmffl&coi&^mmmcQm&izii. m. 
±mmi%iz±z Ktth^m&tmwx^ mxm. » 
/h<a. mmmn&mtzm-&tim^m<r)W?cod*>. m. 
Am<v>b>z%mi^mwz^mmmz*Lx^&bm 30 
feztih. zotn^ Amizxmmtz&^xvifct?, 
Amm%him£xi>. -^&±<^tm^mmiML 
zmvnzn%x-zz>. ztuzi:*). mmmzmizx 
m%bxi3cwi®tfmix^t:i$f&iz. mtt^z 

b L1zA<V$feb „ mtt^X'i&^b LfeAofiJ^ 

amtmzjttug. mwzfiitiAmz x *> xmsmi 

z^w. mxm5sm<v$imz£titfztitzmt>ii 40 

[ 0 0 1 9) rnmsizxtm. zm&&wk 
■*-*mc. \m&mz^x. ^<v$mm.T-9b 
-^nmmniT-fobnfflx: mtr^bi-fzA 

mtt<ZX*j:^bLfi:A&.btfimZti, Wt 
<K*££fc^?VvC. Ztit,<?>A®ffm.X'b?>%&lz 

ti. 3mmi¥-mz&*). w^^mimj 
nsifc-c. 'm#%^i%£X'i>mmm i mf.z 

tih . i 0 » ASfci BMIBW^ b Liz 

Amb.m&^zx'i&^bLtzAmmwcx'h'it: so 
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1 0 

[0020] 

[mam} jht, Mfc^^+iwio-jiitmR 
w-* . 0 1 tt*a?s«* 1 wwtwfcfeH- &mmm 
^mmzwzifctmmx'b 0 . sn^ii c p u *± 

6 . BHcfiwc , 1 HttSiMfll • 

^-;k 2l4*M39NHBISihE> ? *-/k 3ttWfll$| 

h. ft. ZZXlt&^iS^-MZ. ro^5A (V7h 

[0021] n&m&m ■ naffifeHKRjg*^-*' 

[0022] i1^ff«^^- ; e^jL-;i/2»i. ^=at< t 
t>— AULhfcJ: oTff hivfzAmmm<^^ A* L 

t LtzA<m. mmtK^x-^b tfz 
A^ss-fhaw*. 

[0023] mmmm 3.-^31*. mtt** t l 

1zA<m.b . mtfKZX-K^b LizAm.<V®&fr 

[0024] &MBk&ei;*-fr4 n-mtmm ■ 
wm^mr-wm&^^-junzmf^mzir). m 
m^mizj: &mtbAfs<m>mbir~%.Lx^zm£ 

[0025] 4 traSHE^**.— *5tt, 

ti**«fcAIB»)H»i:*«-SUrvvarvi«S-(=. A 

tfmmzm^tzz t izm-z^/i^ < mz&fct& . 

[0026] Wn+Arf-* tW«^a-;P6tt. 

S. 

[0027] iFRgrt y 7 r 7 Id. im^n^^fflSr 
[0 028] V— htyj-^Sli, M^7r7 
[0029] aa^ffijSJ; 0***SBt8f!(c J:*itf. tt 

KtT-^(^fcA^* j ^r 
[0030] ztit>cr>A)}T-9<7)ft. wwm^mk 



1 1 



[oo32]^i evmm,zmhtm<vm.t tx 

T> fctH^fc UtAoASk. Esrr^-c&ok 
LfcA^A&^Jl^^ IBW'^f-J'tfti*'. 

****** k bfcAOA&k&^rr^-e&^k Lfc 
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1 2 

[0033] fflxBLfcXT -/T 1 2 KfcttSElSjrt'^ 
r-^A^a^Mtf^k U\ 8@fft*£ k bfcA 

t^nmx'fo&t^-hmTmix'Z&. mum^ 

<0A A a 0 0 9<7»§£\ ESlfr^ ~C*S k LfcAiHi 
5AT&D. E^*£-C£^kL*:ASctt2AT'&& 
10 vZk*>^ mtf<ZT-9X'i>htW!fc?h. 
[ 0 0 3 4 ] X-f »7 1 2 £ilft*1fc9ltt-t& 
ffl^l^rSt UTtt, MitfAAa 0 0 

-r** t iti Acorns 4 xxh *) . teet^s-ca^k 

U:A<0S#3AT*>6J:3tcu AStiS^JBr*^ 
[®2] 



> 0. 7 



(3) 



[0035] mz-o-^^m^f^t^f-^^-xffi 

tpcD-^-^cryr-flzHLX^t ^.^LTV^ffiffi^ 
r-**\ Xf y 7 1 3 k LX 1 ^ 

3frt>&l>tlXZt:T-9W. *tfc*->T«lftS#ift. 30 

2oI3Lhco«^#ffi* s mffi$^S«^t«. JW<Jcif#«S 
^#&&^£#ia^OA;&T-?a^&i> 9*83 
ti& . mz . Xf y T 1 4 k tTfl^fFffiffi • 

Xfflc^^fS^St. W^3JISi^^ A -^3 
WtaWJMS**!.. 2oc^*<a£U^§-£fcttXx>y7' 
1 5 k LT^e^*^-^ 0. fll* 

^6fcM£. ^.x yri4tt>ftSit^O^, 
ffikW$fflffitv*wM=<kSW£Offik#^&n>T^ 40 

h k § mifietev<-y 7 r 4>o*£*-'t& k 

AWLn&ttimt^X^tLtiABKOmi:. i-i-fr 
y- 4 fW* if*-* 5 fci££> . Xf y 7* 1 6 k LT ^ 

x o . n#^*sk w^jiffi* s^-* 3 <m&m 

[0036] Xx-y7"16{cfc(tS. ^vkr -f <£>ffr& 

o-mmtt Lxa. : ftbizif>t>ti*i£.%.mzmL-?& 



[0037] mim&rC: &a.-A>4 fc . A H" 

Sty'a-^ 5 X -5 T#£iut8!§g#2;k AScoWBf 

<7>-^5-^rtt«. xr-v7i7tLX¥imcD§m 
ztfiK &m^m£i*<QA®tikm^m<7mm<7>-^$(m 
* mm£Limm&k£s&? s . ifecxf •/ r 1 8 1 l 

[ o o 3 8 ] htittxcomm^mizit xf-/ 
8-i izti^x^-t ^ixtz^x<r>im^mznLx<m 

x v-h^ : JjL-/^8 izts^xmmm(0'i^ ^mizw<. 
tv— h . 94*t9v- ym<nmt<m&xm%,vmL 

X%h. Xx y7*2 0k LTy-ht^i-;l'8 

i k T, AS^fJBf tJSv S i 

kffX'Zh. 

[00393 Xf-/r 1 6 izti^^T)\^r^sm<m 
<T>- j mti&k LX\*. 06tCi3ftSAAal OOkAA 
b 0 0 1 Zimth k , A A a 1 0 0<iJJ^k'?)Aa« 
«^1-^T'=3r^kLTV^-&tti6|^>-r. ^#ffifcJ; 
S^**»SWBfSrLTV^OtcitL. AAbOOl 

tiie^^^ k A<vwLkmmt^x%^k 



(8) 
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14 



[0040107 fiZ^t^&iirr A ftg^ 5 
^b LtiAWb i:~mmiztmthtm^ y'7r2 1 

[0041] i^-t/l^f HS*^— ;P5t>(tS» 

LTAaSfifcttSR-*"^ k L/cAScfc^r^-^T^r 20 
v^fc LJtAJSi: 7 t 2 1 fcJSIftU ifcfcXx 

•y T2 6 fc VZffW* 7 T 2 1 * <0 2 OCOt— ? Offi 

$V^^^i'*a-;l'2 3«*? A* 7 7 r 2 3 b 
(C. *#^33<9ffi£WW77 2 3a£l8ttU X 
T77 2 7 b IXftf-Xv 7 T 2 3 b kfttt/W 7 r 2 
3 a^ffi^BfcgCSrfi^ MUf'/r28i:U, ^ 
-f/kr >\x-7l-2 4(ci>v^T^tuttt5r 1 

[0042] ^<r>$mmz£il\f. 06ti>ft6AAa 30 

l o o««fc:A^^JBf* ? »i0rH8:L-cv^^t>i*)^ 
1\ «*f8^BhW»**afctt 1-14-6 = 0. 
83fc&9,AAb001 0W^URK>¥«fcPath.-C» 
1-3^-4 = 0. 2 5fc*Sfc / **Vl'^* 

^(^imt^-^Lx^hizh^-rm^^mm^-r^- 

£31 



[0043] *%HJ{cfctt^^;l^r a <tt»£.3rii. 0 
6fcfctf&AAa 1 0 OcoaitC. »$fc^k'<^A# ^ 8^§^i-r 
**T*v»kL-CV**fcM*&"f, UBH^ttfciiti* 
*Wflr4«R*l/OtfUSf, i^#£a { AJgoWffiC- 
SLfcv>kv*3fll«0>aiBB«<. AAb00l<0« 

Mt-fKrjfcS&r&ftvwr. AIStf>WKk-SL«rv* 

b\^o ffi&VSmHi'bZ V> fc *3 £ fc & fc V^d 
ifcfcafcftvois. 

[0044] »xy hnt:-£0«&t: 

itMtt, m 9 izijki-X o icflKfc J: -oTSOIT* S . 0 
9tSrT^^-{|W J Eyj«.-A'5«>-Hli. A*£ 
tifzWitt^ b LtzA®.tmtt^X-*c^t UzA 

7 r 3 1 2 ocOx-^Ma^JWSrff ->£I£A& 

^tt«-ri»»§rAifW ; E>'*i-^3 2 k , mtt^ 
b LtzAWLtmmt^x-^^t LtiAmtimAmt 

**5>a-*3 2*»fe*^*l*JW/Jk*»fexy hot 

Hi£ft*1-6xyhue-ft3r*soL-/i'3 3fc. i 

[0045] WB/rm&t. 0*4^^ H** S** 
-;U5Kfc(t6BUi£9}&gfc LTfcL 010 (£*-*-.): 3 
fc. X^yT3 5fcLTA^$*ut8^K#fcU:A 
3Rk«aWC*-C«r^kLfcA»fc*1HP<v7r3 1 
WSttU W,zXTvr36bLXttm><v7r3\* 

M^coT-fcomzmAmtn-^i?*.-^ 2iz& 

^X&L&bitXfimUcAmimL. iJttxybn 
t:-9m^: : Ja.-tU3 3KiJWCXr »73 7 1 LTJ31 
Tfcj**-<4) J: 9 xy h oe-«WH«»ff**i6 . 

[0046] 
[SC3] 



E=- 



»&tftAa 



log* 



log, 



•(4) 



^Hr^fl0|**yji-^3 4(ci»^T (1-E) Srffr 

AAa 1 0 OcotttiAiaiOfJBf^lff-StLrv^^t 
99 = 0. Olb^^-t^TAcrmiz^ho -<7)S£ 

Afflcwimtmtix\^i%iiiz§.<AmcvpMbmx 



[0047] at. 4&&fam20&mm&tmt& . 

mint, ^mfomiz&ii&mmm^wmimw. 
coffijmx't,*) . mimicpu£±wbLti3>v*- 

50 -?RV7u? r: 7Jxfrt>ffif$.ZtlX^&. 
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1 5 

[0048] &2<rm&m-C\3ffll 1 iz^TX 0 fc. 0 

in-? m&mmmm aa-ts . 
[0049] aa sfifc-r- ?frt>tm^mv>mjj 

-WHJ&e**-A' 1 k . 4*5:< k i>— AULtfcJ: 

#*-s*hoasc<osii^*^. mi^m { mizm~?tiz. 

b Hzm-h^i-)VT 4 m^th^-f-twr 4 tW^e v 

f •< ffl<D¥^£ k h^sr^-f-iVT- < ft** ^' 6 
t. xmz j:&mtf<£ t^t^m.m\^t^ . a 

OfPga^H^1-SflOTfiiliffifm : E^A-^4 2 1 , 30 

m&mtmzLxftm-z>*m>^7TA3k. t>m 

A' 7 7r4 3 *<0&5Sf^££ fHffifik . SfFWIk . 

4<oi offlto^^a— /ua^HHSSiiS. di-Ct. 85 
l^Hife^k #*yi-;Wi (V 

x 7fc J: l.ffiJSfc^III'C&S . 
[0050] A*T-*^&S^1^*£fc§gfS£3c 

bnmmeK*. ztiznmftLfiimwmifemtm 40 
t&vsmwmst* wm-r^xmrn^zMtx- 

[0051] SfS20llMflKm6»9jggk LT 
{±. 012^^3 *-TXx-yr5 1i:tTAS 

mrnksttfa-JMizte^x. k UJt AS 

k ***** -Ctt^rv >k tJt A****!.****. £>*u 50 
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1 6 

BB&fJBrES^-;^ kSIFMftltrei^-rt^ 1 
k . ^JFfififit xjl -^4 2 tM^nS „ 

[0052] mzxf yy°5 2 b Lxmrnqm^ 
)V3\,z&\,vr. mtf^z b LtiK<r>m.t . mtt^ 
tx^bLtihnAwm&frh. mtr^T-? 
xs>ht\ mtf<ZT-*T\i.%^frzmmu « 
ffM^m^mm-^ t LfcA<oat«§g-^<.§-c^ 

[00 5 3] *3t, X-f y7 , 5 2(CtJJt6«i^- / <.#T 

[0054] mz^^cvtimtkfrbT-t'^-x* 

ct>-o— 5©f- ^ tit LT-^-i. ^>ixT 

ffi ■ ^^ffi-^C^JS^^-^ 1 ty'a- 

#Ay7r«. 04fc^rTJ:3tffl«^#^^* 

«*^A*$ttfcT-^t. mmm : ti'j-->v3frt > 

mhitX^tzf-^ifi. ft\iZ%r>xmhZtlh. 201a 

&mikbft&mmmm^<oAfiT-?ffmmt> i )!& 
mztiz. Rtxr^rs 4 b Lxnmtmm ■ 

Muzm-hmkz-mmi&b . 

Wfe^ik^^^. 2 O^*^ L V ^i,Z\±Xy~ -y 7* 
5 5k LT5eatt?££ ; E> r .x-A'4 tfi^&ai 0 . 

[0055]Xf"/754 ttJftSJt^^*. 

^.k^tcti, Sfiffitt«*Ay7r4'^^r^kLt 
ASoffik^r^T^rV^kLfcASt^I^. 
■f ff-g* i^* 5 tiSS . Xf -/ r 5 6 k 

.to. mm&mb®mm*v*-^3<?>mm% 

[0056] Xy-»/75€>1,Z&nh^ 4 Off* 

<ry~$g%&bhx\t. mmmi^mm^zHnh^y 

[0057] Jg&ffi&tESot.-A^ k^;l-r-f ft* 
5 ->X%htitdm^mb AS^JBrco 
-^Jg&^-Tffiti. Xx-/r5 7kLT¥%ffi<Oft*2r 

[00581 SiFflfffiH^N'-v 7 r 4 1 ttSUTfiXx 

77-5 8kLT. m.-t^bLtimm.b. tmt 



(10) 
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1 7 

t-ottsk mme&mxmfsx'* . ®MizmnFcz s 
[oo59] imm&mm't-y ?t42 e&wcfctt 

7- -/T5 9 k LX. mt?^Z k LtiAWimtt. 
irX'ZZ. 

[0060] X7--y76 0kLXSJX£5-£t> 

wzMsmLxcomtmtotmz'&K t&mw&b 
h^-ff>tmtmmxh?><ox\ zzxompmm 

[0061] d^zx'f'/re ittx. m&mbmms 
mtmmmm. timmzm$>tix^&&^b* 

lZtpW<v7T4 3iZtmZtl&. 

[ o o 6 2 ] tttifzzxvtm^mizitL. xf7 

T 5 1 -Xx / 7* 6 1 W:'cr>ymff'{fhtih . x 
^-/T6 1 - 1 tZ&\^X^L^iXtz±X<V8m3&i£.ft 

vx<r>im}mhr>tizb&m'££tiht. 

7-6 2 k LT^SIffM^i^Wl^ 4fci>VvC|fli§ 

[0063] ^SWfifWt i^-^4 414, M&ttB 
1 3fc*f1»«cJtr>TS8RS*i4. ¥i6lifiHW*^ 
,a.-/k4 4<±. ifi|&W7r4 3iKuS6iVC&fc, ff 

BftWI^*««EaajW»'E^a-rt'7 It. fffiilifik 

SAJSffifttt^fr -5 wfflftw*^*-* 72k. mmt 
n^v*-))s7 2X'ttvhixtz®«<n&m'mzftt 

5. TBI. ±R» M<oaa<0«»f»f»a> 

3 k . fm$ft£#1^#3WlH2ffi£ 
V^JHIClfe^T Hl^-t-S y-hti''i-;l'74<7)40 

[0064] ^mmmm^A-^ 4 t^its jaa 

coj&gk l/Ctt, 0 1 AliZnc? J: a 7 r 4 3 
j^iMfikSIilMBI. RWBB»«M£A»S*uWi 
BKBiHireSo.-rt'7 1 tfcwc. XT778 1 k L 
TflUtmML lJSffi£krFO<5>(6)5S£JBvvCfHf 

[0065] 
[ft4] 
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jm = mm- 1 



±m = mm + 



•(5) 



(6) 



S.Xf778 1, 8 2 £^0tft$¥ffiaTK£SCF 

10 r 8 0 fciJ V^T^TeD^^fcoV^TiKfi&tTFffi 

Xf7783i: LTJltift^ tyA-^ 7 2 flcog 
« • JHa&5£SS 7 2 a CJ3V vC „ STM 

a, asm. &Tmmm?>ztt?tiiztti'bz^mizm 

[0066] ifcfcxf- yT-84 k l/C. tt^UffifltW 

»i. ffttifBl 3«3^Jtfi«il1FE^jL-*7 3rt«> 

A-y 7 r(T>m,z5fct i 3 fc. J: oT3o*> 

&.IkA*T£S. 

[0067] TKMa. imsmcoztizticrmt. t 

01 3<0«-&TJ4, #SA»il 3. ¥ffiB»48. 

^ffiD«2 0k^O. ¥aB<o^r*J. M&MUfib 

[0068] i^CXf y 7*8 5 k LT . V- h^iSa.— 
JU7 4 ICJS V^TttdfWftttWhtF V^Jt(C#ffi#^aj* 

§*ts. y-h^>-'A-;i/7 4CiJ<t-&^l8fifflc^ 
[0069] iJcC *f6W^>»3 ^SOtMiiRW-S. 

m-offifcztoz-zztx'. mmmx'b&vx'. 

[0070] £30)HiM£&ft&ttl£HU. 01 5 1: 

S^-J: 3 t^-T ^ k jJ^TS S . 0 1 5 fc^jcffif&fcii^ 

•cot. mm^xvftis^-iwtmmzimmi*:^ 
[0071] g3asgrae&ft*XH<ttistt. n 

6^1-idtc, 05(OXx/7 , l lkXr>yT12«o 
50 ^t^r-/T9 lkLT, ISIgt^-x >y ^^^-^9^: 



(11) 
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19 



i?*-A> 3 <m&r>R.h 0 izMlH- 1 b 

■t^bLtzm. ®m-<$-m%^bixmb& 
tit, 

1 0 0 7 2 ] ^^^^J&HO^JSSrffo^teS^A 

m^mmrc^^iz^ x-f y 7*1 2#n?TSiia. io 
^f-/7i2, 1 3coj?^i{±. mitmmMbmmx'h 

[00 73] 05^-r^x-/T14OftbOt 

3*U 2owffl^Lv^, £L<ti. fUJOT'i&O 
0!l;ttf - 1 a*'* ?7r 4>tf>WftW8rE 3tz£& 
Wfc7y7£5-tt>nX^iz1§r&M*. XrvTl 5b 

brmmimmi*: v 1 o o , mt 20 
k mLxb&wx'znwmz'-mth . 

[0074] ^2<?)HSfe0!l»^^LT^ttcol8lgSr 

mmm^j.-^b. mm>m : f:^^-^3<omz 

mfSfr x •/ 7 1 ^ 9 ZMz. * <I fc -cii^-c^ s . 

ani.fc^-o«ffii»±. sg 1 <nmm<ommzmm-x. v 
?*it*-)V9*Mtizi§&bwmxi>*). ®%£.m& 
x% s nx'*<mm:'-m? & . 30 
coo75] %3<rmmizmh, wmmm. • ses? 

V j.-tU 1 fcirtt&A.;, 7 r *(0f-^ 

[ 0 0 7 6 ] mz. *wmmb*mi\iz i otsaw: * 



20 



I^^V>;fcfc&*l#T-*tf>» : 1 2ffl (1£5g#g> 

1 . and«^ : tmnMt*?&x<?&-y-Y*%f 

[0077] 2. ORfi^ : ^J^^cOV^-ffiA^o 
[0078] 3- fig^:«»t'W>ffl4 > -^ 

[0079]«i^<0a: 9A 
S^-iMfiS! : 12X50 = 600 

{i^-^co-c, ®2a^3wnife0«i. mi<oius« 
<vmfcmiz&vzm?t^ipEtw/j^mii : m&. 

£%&Zbte%\,\ 

[0080] • |g« : l^i&ffofelgmSriaTt^-r. 
[0081 ] 

[*1] 













1. 8% 


5 0.0% 


0. 145 




10 0.0% 


36.5% 


0.3 18 




48. 0% 


9 6.5% 


0. 045 




5 5.3% 


7 3.0% 


0. 076 



m*t>mt>ipt£±oiz^ mmmbm&mTitoRmi 

Aiat j: i mmxumtt^ xK^T—rtm* tix 
b aymx-wmizmttfiE. i.zt> t A^wt^v ^ 

bi>&UZbZ^LX\i?>. Z0>&%. *^£0#a-C 

<mmi. an 



[0082] 

i-ZT-f^-xtm^mzti^x . ^< kt>-AJji 
b. ^m^mz^x%hix^mmmr)f-9bi 



(12) &m¥7 - 2 7 1 7 9 8 

21 2 2 

1 0 0 84 ] ifcfc. ift*JB3 titof. ±E^6mt:Jn 10 [07 ] SfS l ^fSWfcfefta^;^ -f Hftyji- 

SA^t^t^-C^rV^ti-SAib^lsiict^o-C [08] IBl«SdtHfc*H-4^/UT- 

«5BT. fc LfeAftfc . ESJrf-^-C&^fc [09 ] ^f)Vf 4 IUBE $^-/NDte<90fl£ijcr^x 

LfcASfcttKU H8rt*r>fcfrfctfFiM&&6;h*: xAffij£0 

e£&£-f&«IfcT\ ffiftaARfc «k & iHf £«t-? £i§ [Hi 0] ffiO0?<7)^^kf >rH-ft^^A-;HCfc(tS 

[ o o 8 5 ] 4fc, if^4 ^<omm&mv8k miu ^commzmzmmm^mmmm. 

^frt:^rt-Jf-^^I#tl.f-?<-x^ 20 [012 3 ^2<oiaifi^fe:t>»tSfS#«i§g^^fiiii*ffi 

1 „ [014] m 2<nms.mzti^h^mmmn.^^- 

[0086]*^:. mms izxtm % ±m<vmkizm im 1 6 ] m3<mimtzm&ffimm^mmijj& 

ff -5 i§£fc „ 1 , 2 AW/MKWAfc J: -?Tf M2:fifo 30 [017] ^3coHSfi0ffci3JtSM-^fffiffl^'^ 7r<0 

fc^-m, atHflfciu t mcotmmzgtftk t h ^-mmbT-m^^rtm 

*>\ Amtxzmmizm&wmcoAtf. ®m*miz ifft^mwi 

x ixm-ix^Kb Lxh. mmkmgzm'gj- 1 -ftmtmm • *ai¥iSHSffe&E> f .s.->K 2-» 

S^fc^^S. SBHWByi^i?a-yK 3---H«WE> f A--/K 4 

[0087] ^ Jt. a*^6 tifUf . iJBcO^KdJn -&MB&&E£fa-;K 5~"**vkr ■* it**^.*- 
£T. WrHTdAfbWK&fflT-fc 1 ?. V&ttiZk-t 6 7-IHBB/* 

SAafc«^t^T^V^-r-&Ait* i Plgti:^->T »7r, 8- -V-b*>^-/K 

Jt«c#StJ: 0. — o«7)^m«^x- ^ fc 2 1 -fB&K-y^r, 2 2 - it^E^-;K 

*H*T-**>fc ^-C. i: LJtAiitt . «! 2 2 4»^^f i?64tya 

m-^x^bLKAWwimzti. 40 3i-fl*<7 77. 

UimimL*mz£^ftfeibt>tiXim£%&.-t&Z 3 3-.-xyhnt-H«^^ A -;k 3 4-^ 

bx\ meomizxzmB&'n^tii&SitX'hmi^m /^r^mm.^^-^ 4 l-spffiMh^^**- 

[0iBcOfS*^liiH8] 7r, 4 4 ••-^iSTIIifcftS^y a-^. 7 1 •■•©fflKO 

[01 ] xmiom lvmkmkm&ffiwm^m tt»* 7 7 3-i£ 



(13) 



1fPrF7-2 7l798 



mil 



IH2] 











i 

4 




3 





















mat 



















-u7 



[03] 



AND8SR 



AAa € 0 1 


e 


AAa0 0 2 


0 


AAa 0 0 3 


1 


AAa0O4 


0 






AAa099 


0 


AAalOO 


t 


AAalOl 


0 


AAbOO! 


1 


AAb0 0 2 


0 


AAbQOS 


0 







70 :8WT'<arcfcU 



A 
» 

* 




it 

n 

CD 

« 



[04 3 



MSI miff WH9 ww «BW mra Wn 

1©S 2tt£ 30$ 4<0£ 80£ 6<D$ 



AAaOOl 


1 


1 


0 


1 


0 


1 


1 


AU002 


0 


0 


1 


0 


0 


0 


0 


AArfW 


1 


1 


1 




0 


1 


1 


AAaOM 


0 


0 


0 




0 


0 


0 


















AAaOtt 


1 


1 


1 




0 


1 


1 


AAalOO 


0 


0 


0 




0 


0 


0 


AAalOl 


0 


0 


0 




1 


0 


0 


AfeOOl 


0 


1 


1 




0 


1 


0 


AttOQZ 


0 


0 


t 




1 


0 


1 


AAbOOd 


0 


0 


0 




0 


0 


0 



















****** 



[083 



^2 5 



-v2 0 



|ffg^^7r^w<y7rg>tftmi^nearra]^2 7 
* ; 

| 1 frfr, S*ate&g3K» | ^2 8 



(14) 



15BSFP7-271 798 



( mmm& ) 



^1 1 



SOTTAS iUfcA«tt£x timr^vK^t LttA<m <v1 2 




Yes 



1 9 

i 



1 3nT> 



Yes 
1 5 




1 6 



J 



<i-)i>Tj <rmm (mm «f#rao -vi 7 



I 



(15) 



#^¥7-2 71798 



[06] 



[07] 



AAaOOl 


0 


t 


5 


2 


AAa002 


0 


0 


1 


6 


AAa 0 0 3 


1 


1 


o 


1 


AAaO04 


0 


0 


0 


r 












AA&099 


0 


1 


5 


2 


AAa 1 0 0 


1 


0 


t 


« 


AAel 01 


0 


0 


8 


1 


AAbOOl 


1 


e 


3 


4 


AAbO0 2 


0 


1 


4 


3 


AAbO0 3 


0 


0 


0 


7 














I 



T 



^2 3 



T 



1 ^/l^gggfrEi**-* K >2 4" 



[09] 



[010] 



± 



| iv hnlf-WEi?a-* k*3 3 

1 , 

| «&iv?4 fite4n= f i^a 4 



3 



1 2og>te*siT; »gAft«wr^o |^ 3o 



[014] 




83 



I 



r 



| mMSL*W*T&» / v8 4 
.8 2 [ 

*B5 



C_D 



(16) 



*l§iPF7-2 7 1798 



[Sill 



|a«tt«-F3a»»s*g| 





H«4*Br 







I 5 



4 2^ 



■ 1 ■ 





If «« 































4 3^ | »H/f 7 7r I 

4 4nt4 ^ 1 

I ' 



72 



[013] 
-t 



Sfa.-* 



1^ I 



I 







TNUft 


fHtt 




JJttt 




*fcA 


a.22 


1 


aso 


3 


a7s 


a 




0L3B 


3 


141 


2 


M6 


1 




az4 


Z 


ass 


1 




t 




aoo 


4 




4 


ft 6 0 


4 



7 2a> 



El 







*fcA 


7 




B 


«C 


5 




1 2 



1,7 3 



7 4^| y-W*-Jlr |¥aaffittHEES?a-J 



[017] 



AAaOOl 


0 


1 


5 


2 


AAa0 0 2 


0 


0 


1 


0 


AAaOOS 


1 


1 


0 


1 


AAa004 


0 


0 


0 


7 












AAaO 99 


0 


1 


5 


2 


AAal 00 


1 


0 


1 


0 


AAal 0 1 


0 


0 


6 


t 


AAbOO 1 


1 


0 


3 


4 


AAbO0 2 


0 


1 


4 


3 


AAbOO 3 


0 


0 


0 


7 













(17) 



#^7-271798 



5 1 




Yes 



6 2 



5 2 



6 3 



MBM? 



Yes 
5 5 




No 



5 6 
i_ 



J 



< ©*p*j« (SMstio £W*-rSc 



-^5 7 



~v,5 8 



X 



*V-5 9 



^r©ui«rco^r©¥^fi**»5o 



^✓6 0 



i 



(18) 



#gSPF7-27 1798 



[015] 



1 *&mmmm \ 



^isx.—Ar 



J 



I 



1 — 





■uur 



















(19) ^^7-271798 



[0161 



1 2 



;9 1 



Yes 



9 2 



,1 8-1 



iTffl'V. Yes 
^ftffAflStig 
>? 

No 





1 9 




$ 








2 0 







1 S 



9 3 

ft* 

Yes ? 



«^T-C^^LfcA«D»i, ft*r«cft 
» 



7 



I 



-V1 8 



